Pre-data-collection applications test processing system

ABSTRACT

A system and method for determining which of a plurality of pre-data-collection software applications to apply to optically imaged responses received from a response provider.

FIELD OF THE INVENTION

[0001] The present invention relates to a system and method fordetermining which of a plurality of pre-data-collection softwareapplications to apply to optically imaged test question responses in atest processing workflow system that utilizes a plurality of opticalimaging device types.

BACKGROUND OF THE INVENTION

[0002] The scoring of test question responses that cannot be scored bymachine is typically carried out manually or by presentation on acomputer monitor. Manual scoring involves a human manually scoring aphysical test question response sheet. Scoring by presentation to ahuman of the test question responses using a computer involves scoringan electronic representation of the test question response or responsespresented to a scorer via a computer monitor or other machine that canbe programmed to manipulate symbols. It is the latter scoring procedureto which the present invention relates.

[0003] In order to present a test question response to a scorer viewinga computer monitor, several preparation steps typically occur to enablethe scorer to receive the test question response, view it, score it andrecord the score with the necessary precision, speed and accuracyrequired in the test scoring industry. With test processing thatanalyzes optically imaged test question responses, scanners, facsimilemachines and other optical imaging devices known to those skilled in theart are used to create an electronic image of the test question responsethat was provided by a test taker. The electronic images may be brokendown into smaller images generally referred to as image clips. Theelectronic images and image clips may be stored in a computer or otherstorage media known to those skilled in the art. Multiple opticalimaging devices of varying types are often used in a distributed testprocessing system. The electronic images and image clips typically areconverted to data using well known and commercially available opticalcharacter recognition software, image character recognition software andother similar computer programs. The data or images, or both, can thenbe utilized in a number of ways to aid the test scoring process.

[0004] Certain problems arise, however, when a plurality of opticalimaging devices of varying types are used to image test questionresponses. Depending on the type of optical imaging device used (e.g.,intelligent scanner, non-intelligent scanner, facsimile machine, etc.),one or more of a plurality of pre-data-collection software applicationsare typically applied to the electronic images prior to performing datacollection. Distributed test processing workflow systems need a methodto determine which one or more of the plurality of pre-data collectionsoftware applications will be applied to each electronic image.

SUMMARY OF THE INVENTION

[0005] The present pre-data-collection software determination system andmethod determines which of a plurality of pre-data-collection softwareapplications are to be applied to electronic images of test questionresponses depending upon the optical imaging device type used. Anoptical-imaging-device-type command is generated for indicating to aserver subsystem the particular type of optical imaging device beingused to capture, as electronic images, the test question responses. Apre-data-collection software command is also generated for indicating tothe server subsystem which one or more of the plurality ofpre-data-collection software applications is/are to be applied to theelectronic images. The pre-data-collection software andoptical-imaging-device-type commands and a plurality of electronicimages are received in the server subsystem. The server computer thenelectronically selects from the plurality of pre-data-collectionsoftware applications to apply to the plurality of received electronicimages based upon the optical-imaging-device-type command and thepre-data-collection software command. The pre-data-collection softwareapplications are applied to the received electronic images within theserver subsystem system.

BRIEF DESCRIPTION OF THE DRAWINGS

[0006]FIG. 1 is a hardware diagram of a distributed workflow system aspossible using the claimed inventions.

[0007]FIG. 1A is a three tier architecture view exemplary of the typeused in the claimed inventions.

[0008]FIG. 2 is a block diagram of a portion of the network shown inFIG. 1.

[0009]FIG. 3 is a block diagram of the scanning configuration in thenetwork of FIG. 1.

[0010]FIG. 4 is a block diagram of a portion of the network of FIG. 1.

[0011]FIG. 5 is a flow chart of receiving and processing of responseitems.

[0012]FIG. 6 is a flow chart of one embodiment of the workflow trackinginvention.

[0013]FIG. 7 is a flow chart of one embodiment of thepre-data-collection applications invention.

[0014]FIGS. 8, 8A, 8B is a flow chart of one embodiment of theverification invention.

[0015]FIG. 9 is a flow chart of one embodiment of the multiple serverresponse processing invention.

[0016]FIG. 9A is a block diagram of one embodiment of the multipleserver response processing invention.

[0017]FIG. 10 is a flow chart of one embodiment of the image archiveinvention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0018] General Overview

[0019] The present invention pertains to a response document processingworkflow system. Response document processing is often employed in thetest grading and survey analysis industries. By way of explanation, butnot limitation, a need has developed in the response document processingindustry to process more documents, faster, with better accuracy andless expensively than ever before. Electronic imaging has become verypopular in the response document processing industry given theflexibility, speed and efficiency gained through the use of electronicimages over paper documents. Scanners are one popular type of opticalimaging device; another is the facsimile machine. The increased volumeof documents to be processed, and the quicker turn-around time, requiresthe combination of multiple scanners, multiple processing units andmultiple operator stations working in parallel or in series to quicklyfinish a processing job. A network is often employed to couple thevarious hardware components. The networked response processing workflowsystem described below satisfies the above described needs.

[0020] The beginning of a response processing project or “job” may occurwell before the documents containing the responses are received at aresponse processing center. For example, customer or respondentinformation such as name, mailing address, identification number, andother personalized information may be printed onto response forms, suchas test forms or survey forms. The personalized forms may then beprovided to a customer or directly to a response provider with orwithout a personalized cover letter. The above is accomplished bypopulating and maintaining a database or other electronic storage systemwith personalized information necessary for personalization of theresponse forms.

[0021] Whether the response documents received at the responseprocessing center are those that were personalized or not, it iscritical that all documents received are processed and accounted for. Anaudit trail is therefore started at document receipt time. Theprocessing center will collect the documents in pre-selected quantitiesor batches. A batch header document containing a unique bar-code batchnumber or other similar machine readable identifier will be assigned toeach batch of documents.

[0022] The documents will be optically imaged at the processing centerin one batch or subdivided into smaller batches and imaged sequentially.Multiple processing centers may be used to meet customer needs. Whilescanning is the preferred method of optical imaging, facsimile and otheroptical imaging devices known to those skilled in the art may be used tocreate an electronic image. After imaging, the physical documentstypically are stored in a warehouse, their location identified throughthe unique bar-code batch number or other machine readable identifier onthe batch header.

[0023] Several steps may be required prior to collection of data fromthe electronic image taken from each document. For example, imageenhancement and image identification are available applications that maybe implemented prior to data collection. Data collection or recognitionis accomplished using one or more recognition technologies, for which aninterface is provided in the system. The system must be flexible enoughto handle responses in many different languages and disciplines. Becauseaccuracy is critical in testing and response imaging, human verificationmay be applied to any data derived from electronic images that isinvalid or uncertain. The customer may require multiple verification. Insuch a case, if the system detects a discrepancy between the values ofthe verified data, the image and the data values are automatically sentto a third adjudicator for resolution (adjudication). In an alternativeembodiment, a double adjudication embodiment, the data values areautomatically sent to a third and fourth adjudicator for resolution. Ifthe adjudicators disagree, the image is sent to a manager or other NCSofficial for final determination.

[0024] After the data has been verified, it may be sent to other systemsfor processing consistent with tests, such as grading; surveys, such astallying; or other processes known to those skilled in the art.

[0025] Although electronic storage capabilities have helped loweroperating costs, the need remains to minimize the cost of storingelectronic images. An image archive component of the workflow system isprovided. It can be appreciated that the archives will play an importantrole in the event that a dispute arises over the accuracy of data froman imaged response or test. Further, imaged archives can be accessed bycustomers wanting to view the original scanned document for any numberof reasons.

[0026] Hardware Overview

[0027]FIG. 1 illustrates an example of a hardware configuration for thepresent response document processing system. This configuration is shownas an example only; many different hardware configurations are availablefor implementing the response processing functions described below, andothers will become available as hardware technology advances, asrecognized by those skilled in the art. The network shown is generallydescribed as a three tier or three layer architecture, (See FIG 1A formore detail on three tier architecture) including a bottom tier or datalayer of at least one server computer. The data layer may include aserver subsystem including several servers as shown in FIG. 1 at 10-13.Separate servers may be used for such tasks as maintaining a reportdatabase 10, a data capture database 11, a local area network server 12,archived images database 13 and a “jukebox” 14 —a hardware mechanism forallowing access to one of a group of discs, especially CD-ROMs or otherstorage media. The data layer of the hardware system is interfaced witha middle tier or business objects layer.

[0028] The business objects layer provides process management wherebusiness logic and rules are executed and can accommodate hundreds ofusers (as compared to only 100 users with a two tier architecture—alsousable in the present system) by providing functions such as queuing,application execution, and database staging. In FIG. 1 the businessobjects layer includes a site server 15 interfaced through a hubcomputer and a site server 16 to application servers 17 and 18. Theapplication servers in the preferred embodiment utilize Microsoft'sTransactional Server software. The business objects layer interfacesthrough hub computer 20 to a presentation layer or top tier. Thepresentation layer of the preferred embodiment includes computers 21interfaced to the system via a local area network or LAN, a computer 22and a router 23 interfaced to the system via a wide area network or WANand a computer 24 interfaced to the system via the Internet or otherglobal computer network. The internet connection also includes afirewall 25 and router 26. The optical imaging devices 27 (a scanner andfacsimile machine) are shown in this embodiment coupled to clientcomputers 21. In an alternative embodiment, the optical imaging devicesshown, a scanner or facsimile machine 27 may be coupled with computers22 or computer 24.

[0029] The system uses the optical imaging devices 27, usually scanners,in the presentation layer to optically image responses provided byrespondents on questionnaires, surveys, tests or other similar responsedocuments on which responses are received. Theses response documents maycomprise, for example, test forms with “bubbles” or ovals representingpossible answers to test questions, handwritten essays, surveyresponses, or other various types of written or printed or markedinformation provided by the respondent. After receiving the opticallyimaged responses, the system can prepare those electronic images forsubsequent processing, such as test grading or survey analysis. Aresponse item is, therefore, an electronic representation of at least aportion of a questionnaire, legal document or response document,including, but not limited to, a test form with answers, a contract or asurvey. The system and methods described below are then employed toprocess the response items such that they are prepared for subsequentonline analysis. See, for example, U.S. Pat. Nos. 5,752,836; 5,735,694;5,718,591; 5,716,213; 5,709,551; 5,690,497; 5,321,611; 5,458,493;5,433,615; 5,437,554; 5,466,159; 5,452,379; 5,672,060; 5,987,149; and5,558,521 describing online training, scoring, teaching and reportingsystems and methods. See also, U.S. Pat. Nos. 5,711,673; 5,420407;5,262,943; 5,262,624; 5,184,003; 5,134,669; 5,086,215; 4,934,684;4,857,715; and 4,217,487. The patents listed above, specifically theirspecifications and drawings, are incorporated herein by reference tofurther supplement and explain the present invention.

[0030] The system must be capable of supporting various types of opticalimaging devices, such as intelligent scanners like the NCS 5000i andnon-intelligent scanners that utilize the Kofax/ISIS/TWAIN interface.The client subsystem computers 21, 22 and 24 typically initially receivethe response items and may distribute them to the business objects layerserver subsystem before sending them to the data layer server subsystemfor storage, archival or other processing. For example, a responsedocument optically imaged by the scanner 27 will be assigned a uniquefile name and digital identifier at the client subsystem computers 21,22 or 24, then sent via the business objects layer server subsystem tothe data layer server subsystem for storage and further processing usingthe system and methods described below. It is also possible for servers10-13 to transfer the response items via a global, local or othernetwork known to those skilled in the art to another server subsystem ina remote location. The ability to share the response items with anotherserver subsystem makes work sharing and efficient use of resourcesavailable and practical. The servers may be in the same physical area asone another or separated by great distances, such as between differentcountries or continents. For example, the advantages of different timezones can be utilized using multiple servers and work sharing so as toavoid down time and delay in processing the customer's project.

[0031]FIG. 2 is a more detailed block diagram of a portion of thenetwork shown in FIG. 1. As shown in FIG. 2, the optical imaging device27 shown in FIG. 1, is typically a scanner 27 interfaced to a computer28 or client subsystem computers 21, 22 or 24. The scanner 27 contains acamera 29 for reading through optically imaging all or portions of aresponse document. For example, using “clipping” to read individualresponses on a response documents or reading all the responses includingall or some of the document background of the response document (fullpage imaging).

[0032]FIG. 3 shows a more detailed block diagram of a typical scanningunit. Some scanners have camera optics and filter configurations thatare optically “blind” to certain colors, typically called drop-outcolors. As the scanner cannot see these colors and normally these colorsconstitute the response form background or template that the scannerwill not capture the background of the form within the electronic image.

[0033] One or more client computers 21, 22 or 24, preferably implementedwith the most current high performance computer chip and memory, usuallycontain a frame buffer 251 for receiving the scanned image data from thescanner 27. The computer 28, which is optional, is typically interfacedto the scanner 27 and client computers 21, 22 or 24 for controlling theoperation of the optical imaging device, in this case the scanner 27.The system may alternatively be configured such that all of thefunctionality of the computer 28 is with client computers 21, 22 or 24.The computer 28, if used, controls the scanner and thus controls whenimage data is scanned in and subsequently transferred to clientcomputers 21, 22 or 24. The client computers essentially act as a bufferfor temporarily holding the image data. The client computers can also“clip” areas of interest from the electronic image. Clipping involveselectronically removing, typically in software, a portion of theresponse item or scanned image. These clipped areas may comprise anyportion of a response item: e.g., a handwritten essay or selectedresponse positions. For example, if scanner 27 is an intelligent scannersuch as the NCS 5000i, “clipping” of an area of interest from the imageduring scan time and representing at least a portion of the scannedresponse is possible. Examples of two systems for storing and extractinginformation from scanned images of test answer sheets are shown in U.S.Pat. Nos. 5,134,669 and 5,103,490, both of which are assigned toNational Computer Systems, Inc. and are incorporated herein by referenceas if fully set forth.

[0034]FIG. 4 is a block diagram of the hardware and software functionsin a server in the network of FIG. 1. A scan control module 31interfaces with client computers 21, 22 or 24 and receives the imagedata. The image data is stored in a raw item database 36. For example, aSQL compliant industry standard relational database such as MicrosoftSQL Server, may be used. The system typically further utilizes opensystems architecture such as ODBC, to provide the database accessmechanism. In an alternative embodiment, an entry level database isutilized thus removing the cost of a database such as Microsoft SQLServer. The central application repository (CAR) 33 typically storesdocument definitions and handling criteria. The document process queue37 function as a buffer into a main processing module 45 preferably inservers 10-13. The main processing module 45 controls the processing ofresponse items. It controls the transmission of response items to clientcomputers 21, 22 and 24.

[0035] Software Overview

[0036] Although many programming languages and classes may be utilizedto carry out the present inventions, one of the preferred embodimentsuses object oriented programming. Object oriented programming includeswriting programs in one of a class of programming languages andtechniques based on the concept of an “object” which is a data structureencapsulated with a set of routines, called “methods” which operate onthe data. (See FIG. 1A). Operations on the data can only be performedvia these methods, which are common to all objects that are instances ofa particular “class.” Thus the interface to objects is well defined, andallows the code implementing the methods to be changed so long as theinterface remains the same. For example, an Unknown interface is shownin FIG. 1A. Each class is a separate module and has a position in aclass hierarchy. Methods or code in one class can be passed down thehierarchy to a subclass or inherited from a superclass. Procedure callsare described in terms of message passing. A message names a method andmay optionally include other arguments. When a message is sent to anobject, the method is looked up in the object's class to find out how toperform that operation on the given object. If the method is not definedfor the object's class, it is looked for in its superclass and so on upthe class hierarchy until it is found or there is no higher superclass.Procedure calls always return a result object, which may be an error, asin the case where no superclass defines the requested method. FIG. 1Arepresents one overview of the object oriented programming structure ofthe present invention. Others object oriented programming structures arecontemplated and utilized.

[0037]FIG. 5 is a flowchart of a general overview of typical scanningand processing of response forms and responses. The document processingcenter receives the response sheets or other documents, at step 50 andperforms initial clerical preparation of the documents at step 51 priorto scanning. For example, if the documents received include a digitalidentifier in the form of a bar code or other machine readable format,the digital identifier is read into the system using a barcode reader orother similar technology known to one skilled in the art. In analternative embodiment, the documents received do not include a digitalidentifier. In the latter scenario, the receiver of the documents maymanually organize the received documents into predefined groups orbatches. Such batches are a collection of documents, or the electronicimages and/or data derived from the documents, organized in somerelational manner. For example, a university customer may request thatall response sheets from freshman test takers comprise a batch separatefrom the response sheets from sophomores. Batches may includesub-batches. For example, the freshman batch may be further divided bythe sex of the student/respondent.

[0038] The system at step 52 scans the documents comprising one batch ora sub-set of one batch. The system is designed to accommodate varioustypes of optical imaging devices, as disclosed above. If the scanner iscapable, the system may process the OMR bubbles at step 53, during scantime, and store the data derived from “reading” the OMR marks in thework-in-process storage (WIP) at step 54. The system at step 56 can“clip” areas of interest from the document, again during scan time,saving data derived from the image in the WIP 54. The clipped image issaved as an image file in server 10 or client computer 12. The step of“clipping” involves electronically copying a portion of the scannedimage. This can be done at scan time using an intelligent scanner orlater if a non-intelligent scanner is used. These “clipped” areas maycomprise any portion of a response sheet; for example a handwrittenessay, machine printed name or graphical response such as a mark ordrawing. The system may also receive image data directly from foreignsources, magnetic or electronic, and store the data in raw item database36. Subsequent operations on the data are the same regardless as to thesource of the data. After “clipping” areas of interest from the image,the system stores the response items at step 57 in the WIP 54 ortransfers the response items to the server 10 for storage.

[0039] The system can run on any computer using a standard operatingsystem such as Microsoft Windows or NT. The server subsystem typicallyoperates Microsoft Transaction Server (MTS) and Microsoft InternetInformation Server (IIS 4.0) connecting through ActiveX Data AccessObject (ADO) to connect to either MS SQL Server or MS Access (on NTServer) or Oracle (on NT or UNIX) for the database.

[0040] It has been found that if the scanner 27 coupled to the clientsubsystem is an intelligent scanner such as the NCS 5000I, the clientcomputer PC runs well using Windows NT. For programming and supportefficiency, a standard object oriented programming language such as C++,MFC or Visual Basic are recommended. However, one skilled in the artwill appreciate that almost any programming language can be used toaccomplish the present invention. It preferably is also part of thepresent software to utilize the Internet as well as private Intranetsand the flexibility of browsers such as Netscape Navigator and MicrosoftInternet Explorer.

[0041] Prior to providing the electronic images to human resolvers foronline grading or analysis, steps 58-63, the system performs additionalpreparation processes, described in detail below, to ensure the identityof the image, the quality of the image to be presented, the relationshipof the image to the original document from which it was imaged, theaccuracy of data derived from the document imaged and the archivalstorage of the images for future use or reference or both.

[0042] Response Processing Workflow Tracking System And Method

[0043]FIG. 6 is a flowchart of a preferred embodiment of the responseitem workflow tracking and organization invention.

[0044] A human administrator, in steps 601, 602 and 603, inputsinformation into a system database via a user interface. In step 601,information about the optical imaging device(s) to be used in imagingthe response documents is entered. Such information may include opticalimaging device identifier and interface type. Information pertaining tothe document types to be imaged is entered in step 602. Documentinformation is preferably obtained by capturing an electronic image of ablank response form and a combination of commercially available andbespoke software to define document attributes such as regions ofinterest. In step 603 information specific to the particular job orproject to be undertaken, including, but not limited to, a narrativeabout the customer, workflow rules, customer logic and machine readableidentifier, is entered by the human administrator. The above steps can,but are not required to, occur before receipt of the response documentsprovided by respondents which occurs at 604.

[0045] The documents received, and to be imaged, are assembled by thehuman administrator, step 605, into batches or groups of a suitable sizefor processing consistent with the customer's needs. In an alternative,batches may be defined by the customer and if of a non-suitable size,may be divided into smaller sub-batches to facilitate imaging. In step606, it is determined whether the documents received at the processingcenter include a machine readable identifier (“MRI”). A MRI may includea bar code or other similar machine readable number or code. If a MRI isincluded on the documents to be imaged, the administrator will inputbatch information into the system, step 607, excluding the number ofdocuments in the batch. Batch information may include text including abatch name, an optional batch name provided by the customer and anoptional narrative text string. Using a bar code “reader” or othersimilar device, the MRIs will be read and entered into the systemdatabase at step 608. In step 609, the documents are counted using themachine readable identifier and the number of documents to be imaged isstored in a system database. The system generates, at step 611, a uniquebar-code batch identifier for each batch assembled and prints a batchheader document that includes the unique bar-code batch identifier aswell as the batch information described above.

[0046] If the documents do not include a machine readable identifier,step 610, a human administrator manually counts and inputs the number ofdocuments in the batch to be imaged as well as the batch informationdescribed above. The system generates, at step 611, a unique bar-codebatch identifier for each batch assembled and prints a batch headerdocument that includes the unique bar-code batch identifier as well asthe batch information described above.

[0047] In step 612, the documents are optically imaged using scanner,facsimile machine or other optical imaging device known to those skilledin the art. Scanners can be grouped in two general categories:intelligent and non-intelligent scanners. It is a significant aspect ofthe system and method described herein to be adaptable to a variety ofoptical imaging devices, including scanners. If an intelligent scanner,such as the NCS 5000i, is used to electronically image the documents,the system will “clip” data or images, step 613, from certain regions ofinterest on the response document. Preferably, only the data or responseimage is captured, and the response document background is not captured.More than one “clip” may be taken from a document. In step 613, onemultiple image file containing all the image clips from one document iscreated. In one preferred embodiment, the images are in tag imaged fileformat (TIFF). A custom tag typically is utilized that contains anindividual digital identifier (IDI) that uniquely identifies the imageclip. The IDI contains the information about the image that the systemrequires for subsequent display in the correct context. Typical itemsincluded in the IDI are position coordinates and form identification.The system then stores any data derived from the image clips in adatabase, step 614. Storing the IDI in the tag of the image, rather thanin an associated database, permits faster retrieval of the data when theimage needs to be displayed. It has the added benefit of avoiding anyrisk of the image becoming separated from the relevant data.

[0048] If a non-intelligent scanner is used, the scanner typicallyproduces an image file for each page of the document; one image per pageside, step 615. Interpretation of the data typically is not performed bya non-intelligent scanner. In step 616, the image is stored in files, aseparate directory is created for each batch of documents.

[0049] Pre-Data Collection Preparation Of Responses

[0050]FIG. 7 is a flowchart of the preferred embodiment of the pre-datacollection system and method applied to electronic images of responsesin the processing workflow system. The pre-data-collection applicationsare used to effect various tasks upon the electronic images desired forsubsequent data collection. For example, an electronic image of aresponse may require “cleaning” via commercially available software toimprove the electronic image from which data will be captured. In step701 a human administrator generates an image-device-type commandinforming the system of the type of optical imaging device to be used tocreate electronic images of the responses. Optical imaging devicesinclude scanners, facsimile machines and other devices known to thoseskilled in the art that can convert an image into binary data. Thispreferred embodiment utilizes an optical imaging device that outputsbinary data in a tagged image file format (TIFF). Upon selecting anoptical imaging device type in step 702, the system receives a commandinforming it of the optical image device type to be used toelectronically capture the test question responses as electronic images.The word “command” is meant in the broadest possible interpretation toinclude a character string that tells a program to perform a specificaction or supply input information. Specifically, the word “command”includes menu items as commands.

[0051] In step 703, the human administrator is asked to select which ofa number of pre-data-collection software applications are to be appliedto the electronic images of test question responses. Pre-data-collectionsoftware includes any software application that may be applied to anelectronic image. In the preferred embodiment, pre-data-collectionsoftware applications include software programs with the ability torecognize handwriting, software programs with the ability to read abarcode, software applications with the ability to identify an image asbelonging to a particular set of images, and software applications withthe ability to remove from electronic images noise and superfluous lineswithout removing characters, as well as other software applications knowto those skilled in the art of digitizing an optical image into anelectronic image represented as binary data. Upon selection of the same,the system is updated via a pre-data-collection command informing it ofthe pre-data-collection software applications to be applied.

[0052] In step 704, the system begins to receive electronic images ofthe test question responses via the optical imaging device. The outputof the optical imaging device is binary data and preferably, binary datacapable of storage in tagged image file format (TIFF). The electronicimages of test question responses are received in the server subsystemat step 705. The system then determines which one or more of theplurality of pre-data-collection software applications to apply to thereceived electronic images based upon theoptical-imaging-device-type-command and the pre-data-collection softwarecommand, step 706. In this described preferred embodiment, afterdetermining which of the plurality of pre-data-collection softwareapplications to apply to the electronic images, the received electronicimages are returned to the client computer system at step 707. Theselected pre-data-collection software applications are appliedsequentially to the returned electronic images in step 708.

[0053] Typically, the output from applying one or more of the pluralityof pre-data-collection software applications will be stored in theserver subsystem, step 709. For example, an electronic image that hasbeen cleaned of noise and superfluous lines without removing actualcharacters will be saved as an electronic image separate from theelectronic image to which the pre-data-collection software applicationwas applied. Similarly, the data returned from reading a barcode oridentifying an image as belonging to a particular set of images would besaved in the server subsystem.

[0054] Following application of the pre-data-collection softwareprograms, in step 710, one or more data collection software applicationstypically then are applied to the returned electronic images. Datacollection software applications include intelligent characterrecognition, optical character recognition and other softwareapplications that recognize printed or written characters by a computer.Such programs are well know to those skilled in the art of imageprocessing and character recognition. The text file or other similardata generated from the application of data collection software isstored in the server subsystem at step 711.

[0055] In an alternative embodiment, the application of thepre-data-collection software program occurs in the server subsystemrather than the client computer system. Such an embodiment saves thestep of returning the received electronic images to the client computersystem; however, any benefit is offset by the usage of the serversubsystem resources to apply the pre-data-collection softwareapplications in the server subsystem.

[0056] Test Question Response Verification System And Method

[0057]FIGS. 8 and 8A are flowcharts of one preferred embodiment of atest question response verification system and method. Typicallyelectronic images of test question responses can be divided into imageclips, where an image clip is an electronic image smaller than afull-page image. Data is then typically converted from a binaryrepresentation of the image clip using image recognition techniques suchas intelligent character recognition, optical character recognition orother character recognition and image processing programs know to thoseskilled in the art. As recognized by those skilled in the art, characterrecognition involves complex image processing algorithms and rarelyachieves 100 percent accuracy. Accordingly proofreading or verificationis recommended.

[0058] The system at steps 801 and 802 determines if data derived fromelectronic images having undergone character recognition requiresverification. At step 803, the system has determined that verificationis needed and contemporaneously determines how many times each item mustbe independently verified. Typically a human administer pre-determinesthe number of independent verifications any data requires. The systemcan be programmed to apply one or more verification passes to particularitems. Each verification pass is an independent verification of theaccuracy of the data converted from an electronic image. If only oneverification pass is to be carried out, the system retrieves the data tobe verified from the server subsystem at step 804 and makes the dataavailable in a work queue at step 809.

[0059] If more than one verification pass is required, in addition toretrieving the data to be verified and placing the same in a work queueat steps 804 and 809, the system creates an empty field for eachverification pass greater than one at step 805. For example, if threeverification passes are required of a particular item, the system willcreate two empty fields for verification passes 2 and 3. The system thenretrieves an image of the information at step 806 that was convertedinto data at step 804. The system sends the images to work queues, onequeue per each verification pass at step 809. The system then holds thedata and images in work queues at step 809 until an editor logs into theserver subsystem.

[0060] Step 820 of the system provides to a human editor a list ofoptions available from which to select using a graphical user interface.One of the selections available includes verification. The human editorlogs on to the system at step 821. At step 822 the system determines ifthe login procedure has been complied with by the human editor. If not,at step 823, the human editor is logged out of the system. If the loginis accepted, at step 824, the editor is presented with a choice ofparticular editing jobs from which to select. A job, for example, may bea particular set of tests or surveys or questionnaires provided to NCSfor processing. The jobs presented to the human editor are limited tothose matching information known with the particular human editor. Oncethe particular job is selected, the system processes information relatedto each image clip against information pertaining to the particularskills of the human editor. In step 826, the system determines whetherthe human editor will be verifying an item undergoing only one pass ofverification at step 826. If the item is only undergoing one pass ofverification, at step 827, the system presents to the human editorselected image clips recompiled into coherent image portions such thatthe human editor sees on his or her computer monitor the completebackground of a test form but only the selected images clipscorresponding to material necessary for verification, typically theselected image clip and the corresponding data.

[0061] If the human editor is verifying an item undergoing more than oneverification pass, the system at step 828 presents an image only to beverified. The human editor then records a decision based on his or herobservation of the accuracy of the material presented in step 829.

[0062] In step 831 the data verified is submitted to a compare queue.The system determines the number of verification passes the dataunderwent at step 832. If the data was subject to only one verificationpass, then there is nothing with which to compare it and the data ismarked complete at step 839. If however the data was subject to morethan one verification pass, then all the decisions entered by the humaneditor in step 829 are compared in step 835. In step 836, the systemdetermines if all the decisions entered by the human editors are thesame, and if they are the same the data is marked complete at step 839.If however, the system determines that all the decisions entered by thehuman editors are not the same at step 836, the system routes allentered decisions to an adjudication process.

[0063] Multiple Server Processing Workflow System And Method

[0064]FIG. 9 is a flowchart that describes the multiple server testprocessing workflow system and method. At step 901 documents containingtest question responses are optically imaged. The optically imaged testquestion responses are received by a server computer, at step 902,coupled to the client computer system that includes the optical imagingdevice. In an alternative embodiment, the server or servers that receivethe optically imaged test question responses may not be the servercomputer coupled to the client computer that includes the opticalimaging device. The receiving server computer may or may not storeelectronic images. In step 903 the system distributes from the receivingserver computer to at least a second one of the plurality of servercomputers all or a portion of the received electronic images. The servercomputer or computers to which the received electronic images aredistributed may or may not store the electronic images. In step 904 theelectronic images are analyzed in one or more of the receiving servercomputers. It is further contemplated that the analysis step may occurnot only in one of or all of the server computers but also may occur inone of or all of the client computers coupled to a particular servercomputer. The computer or computers which perform the analysis of theelectronic images may return to the work flow system results forcollation with other test question response analysis results, see step905.

[0065] In an alternative embodiment, the receiving server computers areconfigured to analyze particular test response subject matter includingparticular human languages or particular fields of study such as math,science or subjects that may benefit from test processing personnelhaving particular skills related to the subject matters.

[0066] Archiving Electronic Images Of Responses

[0067]FIG. 10 is a flowchart of the preferred embodiment of the testworkflow image archive invention. In step 1001, an optical image of thetest form template is created and stored in the server subsystem. Instep 1002, the optical imaging device “clips” selected portions of atest question response from a test taken by a test taker generating anelectronic representations of the test question responses. Typically,the output of the optical imaging device is in tagged image file format(TIFF); however, other formats may be used such as JPEG, BMP, etc. Instep 1003, the server subsystem coupled to the optical imaging devicereceives the electronic representation of the test question responsesand stores the same. Typically, the electronic representations of thetest question responses are saved on a magnetic medium such as acomputer disk, magnetic tape or other storage medium known those skilledin the art. To minimize the use of expensive archival resources, thetest form template and electronic representation of test questionresponses are transferred to an archival storage medium and storedseparately, step 1004. The archival storage media includesmagneto-optical disks, optical disks and other storage media know tothose in the art as suitable for archiving electronic images. Anelectronic index is generated in step 1005, where the index links thetest form template to the electronic representations of electronicimages—now both stored on the archival medium. The index is typically arelational database that allows the definition of data structures,storage and retrieval operations and integrity constraints. In such adatabase the data and relations between them are organized in tables. Atable is a collection of records and each record in a table contains thesame fields. Certain fields may be designated as keys, which means thatsearches for specific values of that field will use indexing to speedthem up. The fields typically correspond to fields selected by a humanoperator to correspond with particular areas of interest in the test.

[0068] When a user selects one of the archived items listed on theelectronic index, step 1006, the server subsystem electronically createsat display time to the user an integrated electronic representation ofthe completed test, step 1006, prepared by the test taker by recompilingelectronically the electronic representation of the test form templatewith the electronic images of the segregated test question responses.The user is then able to observer an electronic representation of thetest taken by the test taker, typically on a computer monitor.

[0069] In an alternate embodiment, the recompiling electronically of theelectronic representation of the test form template with the electronicimages of the segregated test question responses occurs prior to storageon the archival medium, step 1015. In step 1016, the recompilation stepis often accomplished in the same server in which the electronic imagesof the segregated test question responses and test form template arestored. Optionally, recompilation occurs in a different server computeror even in a client subsystem. An image compression algorithm, such asJPEG, can be applied to the recompiled integrated electronicrepresentation of the completed test prepared by the test taker, step1017. The integrated electronic representation of the completed testprepared by the test taker is then stored, step 1018, in a storage mediasuitable for archival purposes such as magneto-optical disks, opticaldisks and other storage media known to those in the art. An electronicindex is generated in step 1019, where the electronic index can accessthe integrated electronic representation of the completed test stored onthe archival medium. The index is typically a relational database asdescribed above.

[0070] When a user selects one of the electronic index selections, step1020, the integrated electronic representation of the completed test isaccessed and may be presented to a human observer using a computermonitor. A browser installed on the computer coupled to the monitorviewed by the human observer will be able to display integratedelectronic representation of the completed test and facilitatenavigation from one page to the next if the test has multiple pages.

What is claimed is:
 1. In a computerized test processing workflow systemconfigured for analyzing electronic images of test question responsesprovided by a test taker, the system comprising a server subsystemhaving at least one server computer, a client subsystem having at leastone client computer, and at least one of a plurality of types of opticalimaging devices for electronically capturing the test questionresponses, wherein each particular client computer system has a userinterface, a method for applying at least one of a plurality ofpre-data-collection software applications depending upon the imagingdevice type, the method comprising: a. generating anoptical-imaging-device-type command, wherein theoptical-imaging-device-type command comprises information of theparticular type of optical imaging device being used to capture the testquestion responses as electronic images; b. generating apre-data-collection software command for selecting which one or more ofthe plurality of pre-data-collection software applications to apply tothe electronic images; c. receiving a plurality of electronic imagesfrom the optical imaging device; d. based upon theoptical-imaging-device-type command and the pre-data-collection softwarecommand, electronically selecting the one or more of the plurality ofpre-data-collection software applications to apply to the plurality ofreceived electronic images; and e. applying the selected one or morepre-data collection software applications to the plurality of receivedelectronic images; whereby distributed optical imaging devices ofvarious types coupled to client computers may be effectively employedfor data imaging while permitting effective data collection afterselected pre-data-collection software applications have been appliedaccording to the type of optical imaging device used at any particularimaging site.
 2. The method of claim 1, further comprising the step ofapplying character recognition to the received electronic images.
 3. Themethod of claim 1, further comprising the step of storing in the systeminformation generated from application of the one or more pre-datacollection software applications to the received electronic images. 4.The method of claim 3, wherein the step of storing the informationgenerated from the application of the one or more pre-data collectionsoftware applications comprises information about the identity of theplurality of received electronic images.
 5. The method of claim 3,wherein the step of storing the information generated from theapplication of the one or more pre-data collection software applicationscomprises information about cleaned electronic images of the pluralityof returned electronic images, cleaned electronic images representing anelectronic image that has had removed electronically noise andsuperfluous lines without removing actual characters.
 6. The method ofclaim 1, further comprising the step of applying within the businessobject layer of the architecture the selected one or more pre-datacollection software applications to the plurality of received electronicimages.
 7. The method of claim 1, wherein the step of generating anoptical-imaging-device-type command comprises a command for a devicethat output images in tagged image file format (TIFF).
 8. The method ofclaim 1, wherein the step of generating an optical-imaging-device-typecommand comprises a command for a scanner.
 9. The method of claim 1,wherein generating an optical-imaging-device-type command comprises acommand for a facsimile machine.
 10. The method of claim 1, whereingenerating an optical-imaging-device-type command comprises a commandfor devices that digitize an electronic image represented as binarydata.
 11. The method of claim 1, wherein the step of applying theselected one or more pre-data collection software applications to theplurality of received electronic images comprises applying humanhandwriting recognition technology.
 12. The method of claim 1, whereinthe step of applying the selected one or more pre-data collectionsoftware applications to the plurality of received electronic imagescomprises applying barcode recognition technology.
 13. The method ofclaim 1, wherein the step of applying the selected one or more pre-datacollection software applications to the plurality of received electronicimages comprises applying software having the ability to identify animage as belonging to a particular set of images.
 14. The method ofclaim 1, wherein the step of applying the selected one or more pre-datacollection software applications to the plurality of received electronicimages comprises applying software having the ability to remove from anelectronic image noise or superfluous lines without removing characters.15. The method of claim 1, wherein the step of applying the selected oneor more pre-data collection software applications to the plurality ofreceived electronic images comprises applying optical characterrecognition technology.
 16. The method of claim 1, wherein the step ofgenerating an optical-imaging-device-type command is accomplished usinga user interface.
 17. The method of claim 1, wherein the step ofgenerating a pre-data-collection software command is accomplished by auser through a user interface.
 18. The method of claim 1, wherein thestep of receiving a plurality of electronic images from the opticalimaging device comprises receiving an electronic representation of atleast a portion of the test question response.
 19. In a computerizedtest processing workflow system configured for analyzing electronicimages of test question responses provided by a test taker, the systemcomprising a server subsystem having at least one server computer, aclient subsystem having at least one client computer, and at least oneof a plurality of types of optical imaging devices for electronicallycapturing the test question responses, wherein each particular clientcomputer system has a user interface, a method for applying at least oneof a plurality of pre-data-collection software applications dependingupon the imaging device type, the method comprising: a. generating anoptical-imaging-device-type command at any particular client computerfor indicating to the coupled server subsystem the particular type ofoptical imaging device being used to capture the test question responsesas electronic images; b. generating a pre-data-collection softwarecommand at the particular client computer system for indicating to theserver subsystem which one or more of the plurality ofpre-data-collection software applications to apply to the electronicimages; c. receiving a plurality of electronic images from theparticular client computer system and corresponding optical imagingdevice; d. based upon the optical-imaging-device-type command and thepre-data-collection software command, electronically selecting which oneor more of the plurality of pre-data-collection software applications toapply to the plurality of received electronic images; and e. applying atthe particular client computer system the selected one or more pre-datacollection software applications to the plurality of received electronicimages; whereby distributed optical imaging devices of various typescoupled to client computers may be effectively employed for data imagingwhile permitting effective data collection after selectedpre-data-collection software applications have been applied according tothe type of optical imaging device used at any particular imaging site.20. In a computerized test processing workflow system configured foranalyzing electronic images of test question responses provided by atest taker, the system comprising a server subsystem having at least oneserver computer, a client subsystem having at least one client computer,and at least one of a plurality of types of optical imaging devices forelectronically capturing the test question responses, wherein eachparticular client computer system has a user interface, a method forapplying at least one of a plurality of pre-data-collection softwareapplications depending upon the imaging device type, the methodcomprising: a. receiving at the server subsystem anoptical-imaging-device-type command indicating the particular type ofoptical imaging device being used to capture the test question responsesas electronic images; b. receiving at the server subsystem apre-data-collection software command indicating which one or more of theplurality of pre-data-collection software applications to apply to theelectronic images; c. receiving a plurality of electronic images at theserver subsystem and from the optical imaging device; d. based upon theoptical-imaging-device-type command and the pre-data-collection softwarecommand, electronically selecting which one or more of the plurality ofpre-data-collection software applications to apply to the plurality ofreceived electronic images; and e. applying the selected one or morepre-data collection software applications to the plurality of receivedelectronic images using a server component; whereby distributed opticalimaging devices of various types coupled to client computers may beeffectively employed for data imaging while permitting effective datacollection after selected pre-data-collection software applications havebeen applied according to the type of optical imaging device used at anyparticular imaging site.
 21. In a computerized test processing workflowsystem configured for analyzing electronic images of test questionresponses provided by a test taker, the system comprising a serversubsystem having at least one server computer, a client subsystem havingat least one client computer, and at least one of a plurality of typesof optical imaging devices for electronically capturing the testquestion responses, wherein each particular client computer system has auser interface, a method for applying at least one of a plurality ofpre-data-collection software applications depending upon the type ofscanner, the method comprising: a. generating a scanner command forindicating to the server subsystem the particular type of scanner beingused to capture the test question responses as electronic images; b.generating a pre-data-collection software command for indicating to theserver subsystem which one or more of the plurality ofpre-data-collection software applications to apply to the electronicimages; c. receiving a plurality of electronic images at the serversubsystem and from the scanner; d. based upon the scanner command andthe pre-data-collection software command, electronically selecting whichone or more of the plurality of pre-data-collection softwareapplications to apply to the plurality of received electronic images;and e. applying the selected one or more pre-data collection softwareapplications to the plurality of received electronic images; wherebydistributed scanners of various types coupled to client computers may beeffectively employed for data imaging while permitting effective datacollection after selected pre-data-collection software applications havebeen applied according to the type of scanner used at any particularimaging site.
 22. A system for applying at least one of a plurality ofpre-data-collection software applications depending upon the imagingdevice type in a computerized test processing workflow system configuredfor analyzing electronic images of test question responses provided by atest taker, the system comprising a server subsystem having at least oneserver computer, a client subsystem having at least one client computer,and at least one of a plurality of types of optical imaging devices forelectronically capturing the test question responses, wherein eachparticular client computer system has a user interface, the systemcomprising: a. a processing system configured to generate anoptical-imaging-device-type command, wherein theoptical-imaging-device-type command comprises information of theparticular type of optical imaging device being used to capture the testquestion responses as electronic images; b. the processing system alsobeing configured to generate a pre-data-collection software command forselecting which one or more of the plurality of pre-data-collectionsoftware applications to apply to the electronic images; c. theprocessing system further configured to receive a plurality ofelectronic images from the optical imaging device; d. the processingsystem being further capable of electronically selecting the one or moreof the plurality of pre-data-collection software applications to applyto the plurality of received electronic images based upon theoptical-imaging-device-type command and the pre-data-collection softwarecommand; and e. the processing system also being configured to apply theselected one or more pre-data collection software applications to theplurality of received electronic images; whereby distributed opticalimaging devices of various types coupled to client computers may beeffectively employed for data imaging while permitting effective datacollection after selected pre-data-collection software applications havebeen applied according to the type of optical imaging device used at anyparticular imaging site.
 23. The system of claim 22, further comprisingan electronic storage component capable of storing in the systeminformation generated from application of the one or more pre-datacollection software applications to the received electronic images. 24.The system of claim 22, wherein the processing system further comprisesa server component for applying the selected one or more pre-datacollection software applications to the plurality of received electronicimages.
 25. The system of claim 22, wherein theoptical-imaging-device-type comprises a device that outputs images intagged image file format (TIFF).
 26. The system of claim 22, wherein theoptical-imaging-device-type comprises a scanner.
 27. The system of claim22, wherein the optical-imaging-device-type comprises a facsimilemachine.
 28. The system of claim 22, wherein theoptical-imaging-device-type comprises a device that digitizes anelectronic image represented as binary data.
 29. The system of claim 22,wherein the pre-data collection software comprises human handwritingrecognition technology.
 30. The system of claim 22, wherein the pre-datacollection software comprises barcode recognition technology.
 31. Thesystem of claim 22, wherein the pre-data collection software comprisessoftware having the ability to identify an image as belonging to aparticular set of images.
 32. The system of claim 22, wherein thepre-data collection software comprises software having the ability toremove from an electronic image noise or superfluous lines withoutremoving characters.
 33. The system of claim 22, wherein the pre-datacollection software comprises optical character recognition technology.34. The system of claim 22, wherein a user interface generates theoptical-imaging-device-type command.
 35. A computer program productreadable by a computing system for a computer process for a computerizedtest processing workflow system configured for analyzing electronicimages of test question responses provided by a test taker, the systemcomprising a server subsystem having at least one server computer, aclient subsystem having at least one client computer, and at least oneof a plurality of types of optical imaging devices for electronicallycapturing the test question responses, wherein each particular clientcomputer system has a user interface, a method for applying at least oneof a plurality of pre-data-collection software applications dependingupon the imaging device type, the computer program product comprisingencoded instructions either embedded on a computer-readable media ortransmitted in a propagating wave, the encoded instructions beingarranged and configured to be read by the computing system for causingthe computing system to perform the steps of: a. generating anoptical-imaging-device-type command, wherein theoptical-imaging-device-type command comprises information of theparticular type of optical imaging device being used to capture the testquestion responses as electronic images; b. generating apre-data-collection software command for selecting which one or more ofthe plurality of pre-data-collection software applications to apply tothe electronic images; c. receiving a plurality of electronic imagesfrom the optical imaging device; d. based upon theoptical-imaging-device-type command and the pre-data-collection softwarecommand, electronically selecting the one or more of the plurality ofpre-data-collection software applications to apply to the plurality ofreceived electronic images; and e. applying the selected one or morepre-data collection software applications to the plurality of receivedelectronic images; whereby distributed optical imaging devices ofvarious types coupled to client computers may be effectively employedfor data imaging while permitting effective data collection afterselected pre-data-collection software applications have been appliedaccording to the type of optical imaging device used at any particularimaging site.
 36. A computer program product readable by a computingsystem for a computer process for a computerized test processingworkflow system configured for analyzing electronic images of testquestion responses provided by a test taker, the system comprising aserver subsystem having at least one server computer, a client subsystemhaving at least one client computer, and at least one of a plurality oftypes of optical imaging devices for electronically capturing the testquestion responses, wherein each particular client computer system has auser interface, a method for applying at least one of a plurality ofpre-data-collection software applications depending upon the imagingdevice type, the computer program product comprising encodedinstructions either embedded on a computer-readable media or transmittedin a propagating wave, the encoded instructions being arranged andconfigured to be read by the computing system for causing the computingsystem to perform the steps of: a. generating anoptical-imaging-device-type command at any particular client computerfor indicating to the coupled server subsystem the particular type ofoptical imaging device being used to capture the test question responsesas electronic images; b. generating a pre-data-collection softwarecommand at the particular client computer system for indicating to theserver subsystem which one or more of the plurality ofpre-data-collection software applications to apply to the electronicimages; c. receiving a plurality of electronic images from theparticular client computer system and corresponding optical imagingdevice; d. based upon the optical-imaging-device-type command and thepre-data-collection software command, electronically selecting which oneor more of the plurality of pre-data-collection software applications toapply to the plurality of received electronic images; and e. applying atthe particular client computer system the selected one or more pre-datacollection software applications to the plurality of received electronicimages; whereby distributed optical imaging devices of various typescoupled to client computers may be effectively employed for data imagingwhile permitting effective data collection after selectedpre-data-collection software applications have been applied according tothe type of optical imaging device used at any particular imaging site.37. A system for applying at least one of a plurality ofpre-data-collection software applications depending upon the imagingdevice type in a computerized test processing workflow system configuredfor analyzing electronic images of test question responses provided by atest taker, the system comprising a server subsystem having at least oneserver computer, a client subsystem having at least one client computer,and at least one of a plurality of types of optical imaging devices forelectronically capturing the test question responses, wherein eachparticular client computer system has a user interface, the systemcomprising: a. means for generating an optical-imaging-device-typecommand, wherein the optical-imaging-device-type command comprisesinformation of the particular type of optical imaging device being usedto capture the test question responses as electronic images; b. meansfor generating a pre-data-collection software command for selectingwhich one or more of the plurality of pre-data-collection softwareapplications to apply to the electronic images; c. means for receiving aplurality of electronic images from the optical imaging device; d. meansfor electronically selecting the one or more of the plurality ofpre-data-collection software applications to apply to the plurality ofreceived electronic images based upon the optical-imaging-device-typecommand and the pre-data-collection software command; and e. means forapplying the selected one or more pre-data collection softwareapplications to the plurality of received electronic images; wherebydistributed optical imaging devices of various types coupled to clientcomputers may be effectively employed for data imaging while permittingeffective data collection after selected pre-data-collection softwareapplications have been applied according to the type of optical imagingdevice used at any particular imaging site.